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Abstract: 

The paper studies the structural sensitivity of line pattern 
recognition using shape graphs. We compare the recognition 
performance for four different algorithms. Each algorithm uses a 
set of pairwise geometric attributes and a neighbourhood graph to 
represent the structure of the line patterns. The first algorithm 
uses a pairwise geometric histogram, the second uses a relational 
histogram on the edges of the shape graph, the third compares 
the set of attributes on the edges of the shape graph and the final 
algorithm compares the arrangement of line correspondences 
using graph matching. The different algorithms are compared 
under line deletion, line addition, line fragmentation and line end 
point measurement errors. It is the graph matching algorithm 
which proves to be the most effective 
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Abstract 

This paper studies the structural sensitivity of line- 
pattern recognition using shape-graphs. We compare the 
recognition performance for four different algorithms. Each 
algorithm uses a set of pairwise geometric attributes and a 
neighbourhood graph to represent the structure of the line 
patterns. The first algorithm uses a pairwise geometric his- 
togram, the second uses a relational histogram on the edges 
of the shape graph, the third compares the set of attributes 
on the edges of the shape graph and the final algorithm 
compares the arrangement of line correspondences using 
graph-matching. The different algorithms are compared un- 
der line deletion, line addition, line fragmentation and line 
end-point measurement errors. It is the graph-matching al- 
gorithm which proves to be the most effective. 

1 Introduction 

Graphs have proved to be seductive yet highly elusive 
as representations of shape in computer vision [2, 18, 22], 
They are seductive since they convey the topological ar- 
rangements of features or object primitives in a manner 
which can be invariant to both viewing angle and shape de- 
formation. On the other hand, they have proved to be very 
elusive since the process of eliciting a relational abstraction 
from poor image data is one of extreme fragility [22]. The 
idea of using graphs as representations of 2D scenes can be 
traced back to Barrow and Popplestone [2]. Over the past 
two and half decades, the topic has been the subject of sus- 
tained activity. However, the fact remains that inexact graph 
matching is a difficult problem and that there is no "free 
lunch" on offer. There is a stark choice of either accept- 
ing limited recognition performance or deploying consider- 
able computational resources to get an acceptable answer 
This trade-off has been highlighted by the interest in large 
image databases. If such databases are to be queried us- 
ing high-level relational object descriptions rather than us- 
ing low level feature characteristics, then a fast and reliable 


means of graph-matching is required. 

There are several recent examples of the use of graph- 
based representations for 2D shape recognition. For in- 
stance, the FORMS system of Zhu and Yuille [23] uses 
skeletal shape graphs to model articulated objects. Liu and 
Geiger [13] have taken these ideas further by developing 
a hierarchical model which achieves a degree of unification 
between the articulated shape graph and the detection of raw 
image features via the Mumford-Shah functional [14]. Amit 
and Kong [1] have a MAP framework for modelling 2D 
deformable shape using a decomposable graph representa- 
tion. More recently, there have been several attempts to use 
graph retrieval as a means of recognising 2D shapes from 
databases. Much of this work can be viewed as providing a 
concrete realisation of the ideas introduced by Leyton's [ 1 2] 
process grammar for shape. For instance Siddiqi et al [20] 
have used the shock graph derived from the singularities of 
the reaction-diffusion equation to provide a skeletal repre- 
sentation of 2D binary shapes. Shape recognition is realised 
using the subtree matching algorithm of Reyner [16]. The 
matching process has been refined by Pelillo et al[\S\ who 
establish a mean of matching association trees using a re- 
laxation algorithm to find the maximal clique. 

The observation underpinning this paper is that although 
this work has done much to establish the representational 
expediency of shape graphs, there has been little effort di- 
rected at answering the question of how much structural 
corruption can be tolerated and how recognition accuracy 
is traded against the computational resources expended. 
It is these issues which are the focus of this paper. We 
aim to compare four different strategies for rapid graph- 
matching. The matching algorithms become increasingly 
complex by placing greater reliance on graph-structure in 
the shape recognition process. The application vehicle used 
in our recognition experiments is a demanding one. We 
aim to recognise large line-patterns segmented from grey- 
scale images. The simplest line-matching algorithm uses 
a histogram of pairwise geometric attributes on pairs of 
line-segments. The second algorithm uses the edge-set of 


a shape-graph to gate contributions to the histogram. The 
third method uses the similarity of features-sets defined on 
the graph-edges. The final algorithm uses an iterative pro- 
cess to recover an explicit arrangement of correspondences 
between graphs and retrieves the pattern fi-om the database 
on the basis of maximum consistency. 

Our study is conducted on a database of some 2500 line 
patterns. In order to investigate the structural and attribute 
sensitivity of our matching algorithms we subject the pat- 
terns in the database to four different kinds of segmentation 
error. These are line addition, line deletion, line-splitting 
and segment end-point measurement errors. We measure 
the recognition performance against each of these errors. 
Although the sensitivity pattern is quite subtle, the main 
conclusion of our study is that the iterative graph-matching 
algorithm offers the best overall performance. 

2 Framework 

Formally our recognition problem is posed as follows. 
We abstract the shapes to be recognised as attributed re- 
lational graphs (ARG's). Each ARG in the database is a 
triple, G = {VojEoi Aq), where Vq is the set of ver- 
tices (nodes), Eg is the edge set (Eg C Vg x ^g), and 
Ag is the set of node attributes. In our experimental ex- 
ample, the nodes represent line-structures segmented from 
2D images. The edges are established by computing the 
N-nearest neighbour graph for the line-centres. Each node 
j e V is characterised by a vector of attributes, x^- and 
hence Ag = {^j\3 ^ In the work reported here the 
attribute-vector represents the contents of a normalised pair- 
wise attribute histogram. 

The database of line-patterns is represented by the set of 
ARG's V = {G}. The goal is to retrieve from the database 
X>, the individual ARG that most closely resembles a query 
pattern Q = {Vq,Eq,Aq). We now furnish some details 
of the shape retrieval task used in our experimental eval- 
uation of the recognition method. In particular, we focus 
on the problem of recognising 2D line patterns in a manner 
which is invariant to rotation, translation and scale. The raw 



Figure 1. Geometry for shape representation 


information available for each line segment are its orienta- 
tion (angle with respect to the horizontal axis) and its length 
(see Figure 1). To illustrate how the Euclidean invariant 
pairwise feature attributes are computed, suppose that we 
denote the line segments associated with the nodes indexed 
a and b by the vectors v^ and y^, respectively. We use two 
pairwise attributes. The first is the relative angle given by 


Oa,b = axccos 



The second is the normalised length ratio between the ori- 
ented baseline vector v^ and the vector y/ joining the end 
(b) of the baseline segment (ab) to the intersection of the 
segment pair (cd). 

The two attributes are used as a feature- vector Za^b = 
{Oa.bj ^o.fc)^ for the line-segment pair. 

Each node in the shape graph, i.e. each line in the pat- 
tern, is represented by the histogram of its pairwise geo- 
metric attributes to the remaining lines in the pattern. This 
histogram can be thought of as a local estimate of the prob- 
ability distribution for the pairwise attributes. Accordingly, 
the angle and position attributes Oa,b and -da^b are binned 
in a histogram. Suppose that Saiii^y) = {{O'ib)\0a^b € 
A^ A t9a,& e Ri, A b e Vo} is the set of nodes whose 
pairwise geometric attributes with the node a are spanned 
by the range of directed relative angles A^ and the rela- 
tive position attribute range R,^, The contents of the his- 
togram bin spanning the two attribute ranges is given by 
Hai^^i^) = |5a(/i, i/)|. Each histogram contains rel- 
ative angle bins and ur length ratio bins. The normalised 
geometric histogram bin-entries are computed as follows 

. / V ^a(/i,t/) 

In the remainder of this paper, we use the notation to 
denote the normalised histogram for the graph G from the 
database and to denote the query histogram, 

3 Graph Retrieval Algorithms 

The aim in this paper is to compare four graph-based 
recognition algorithms for retrieving from the database the 
pattern that most closely resembles a query. The four algo- 
rithms use increasingly complex representations. The most 
straightforward uses a global histogram of pairwise geomet- 
ric attributes [8]. The next method refines this idea by us- 
ing only the attributes on the edges of a nearest neighbour 
graph [10]. The third algorithm uses the set of attributes 


on the edges and reaHses comparison on an element-by- 
element basis using a robust error kernel [9]. Finally, we 
use a graph-matching algorithm. Here the similarity of the 
local attribute histograms is used to provide explicit node 
correspondences [7]. The matching algorithm iteratively 
modifies the pattern of correspondences to optimise the a 
posteriori probability of the library patterns given the query 
pattern. Retrieval is realised by identifying the line-pattern 
of largest matching probability, 

3.1 Attribute Histograms 

We commence by considering how pairwise geometric 
histograms can be used for the purposes of recognition [21]. 
The idea here is to conglomerate the node histograms into 
a global histogram [8]. This histogram provides a statistical 
summary for the pairwise attributes residing on the edges of 
a nearest neighbour graph. The normalised histogram bin- 
contents is given by 

The best-matching pattern is retrieved from the database on 
the basis of similarity with the query pattern histogram. Our 
similarity measure is the histogram correlation. The mea- 
sure of pattern correlation is the Bhattacharyya distance. 
The class identity of the retrieved pattern is 

nR 

= argmaxln ^ ^ y u) x u) 

This is the most efficient of our retrieval algorithms. Once 
the histograms have been pre-compiled and normalised, 
then the computational overheads are purely related to the 
number of bin comparisons that must be performed. 

3.2 Relational Histograms 

The next step is to consider how graph-structure can be 
used to compute a compact relational summary [6, 19, 3, 
10]. This method represents a refinement of the global at- 
tribute histogram. Rather then binning the geometric at- 
tributes over all node-pairs, we restrict ourselfs to pairs of 
nodes that are connected by edges in the shape graph. To 
do this, we modify the set of attributes used to represent 
the node a to be those that belong to the first-order graph 
neighbourhood. In other words, we now have 5o(M) ^) = 
{(a,6)|^a,6 e A 'da.b e A (a, 6) e Ed}. The nor- 
malised global histogram can now be computed in the way 
described in the previous subsection of this paper. The ef- 
fect is to use the edges of the shape-graph to gate contribu- 
tions to the histogram. 


The run-time computation is identical to that for the at- 
tribute histogram. However, the amount of precompilation 
required can be less than that for the attribute histogram. 
The reason for this is that once the nearest neighbour graph 
is computed, then fewer angle and relative length attributes 
need to be computed. 

3.3 Feature-sets 

The next step is to consider how set-based representa- 
tion can be used for the purposes of recognition [9]. Here 
we use a variant of Rucklidge's [17] idea of effecting ob- 
ject recognition by using a variant of the Hausdorff distance 
to compare a set of features. We aim to effect retrieval on 
the basis of the similarity of the set of attributes residing on 
the edges of the nearest neighbour graph. The Hausdorff 
distance provides a means of comparing a set of unordered 
observations without having to establish explicit correspon- 
dences between individual elements. However, this mea- 
sure is notoriously susceptible to measurement outliers. For 
this reason, we choose instead to gauge similarity using a 
robust error kernel [9]. The class identity of the retrieved 
pattern is 

where Ta{p) = exp ^— is a robust weighting kernel, 

zf j is a pairwise feature-vector from the query graph and 
zfj is a feature- vector from a target graph. 

This recognition method is more computationally de- 
manding than either of the histogram-based methods since it 
requires each of the set of attributes to be compared. When 
compared with the histogram-based methods, the computa- 
tional effort required is increased by a factor which is pro- 
portional to the average bin contents of the histograms. 

3.4 Graph IMatching 

The feature-based similarity measure does not utilise any 
information concerning the consistency of the arrangement 
of correspondences between the individual elements. In or- 
der to exploit the consistency of correspondences, we use a 
simplification of the graph-matching scheme developed by 
Finch, Wilson and Hancock [5]. This poses the retrieval 
process as one of associating with the query the graph from 
the database that has the largest a posteriori probability of 
match. In other words, the class identity of the graph which 
most closely corresponds to the query is 

iJQ - arg max 

However, since we wish to make a detailed structural com- 
parison of the graphs, rather than comparing their overall 


statistical properties, we must first establish a set of best- 
match correspondences between each ARG in the database 
and the query Q. At iteration n the set of correspon- 
dences between the query Q and the ARG G is a relation 
fo '^G over the vertex sets of the two graphs. The 

mapping function consists of a set of Cartesian pairings be- 
tween the nodes of the two graphs, i.e. /g — {(a, a); o € 
Vg, a € Vq} C Vg X Vq. The retrieved pattern is the one 
which has the most consistent pattern of correspondences 
and satisfies the condition 

{jjQ = arg max rnaxP(/G'|(j', Q) 

The pattern of correspondences is assigned to satisfy the 
following maximum a posteriori probability condition 

/S(a) = arg p{x^,xMgM = <^Wg\Eg,Eq) 
Suppose that we use the notation 


n _ri if/S(a) = a 


' 0 otherwise 

to represent the correspondence assignment. The consis- 
tency of global match against the query pattern can be im- 
proved by iterating the assignment condition 


/S(a) =arg max 


lnp(x„,x„|/a(a) = a)+ Yl 


X: {ln(l - Pe)s2-'s^-/+lnP,{l - sZJs^j')] 

The probability of match between the pattern-vectors is 
computed using the Bhattacharyya coefficient between the 
normalised histograms. 

F(/a(a) = a|x„, x „) ^ exp[-B» ,a] = 

With this modelling ingredient, and using the correspon- 
dence matches delivered by the graph-matching scheme, the 
condition for recognition is 


ujQ = arg max 


51 S i "^'"''^ ~ ^^^^ 
ln(l - PeXa</? + lriP,(l - s2^,sl^)^ 


This is the most computationally demanding of the meth- 
ods. We have found it to be 10 times slower than the 
feature-set method and over a 1000 times slower than the 
histogram-based method. 


4 Sensitivity Analysis 

The aim in this section is to investigate the sensitivity 
of the four graph-based retrieval strategies to the systemat- 
ics of the line-segmentation process. To this end we have 
simulated the segmentation errors that can occur when line- 
segments are extracted fi^om realistic image data. Specifi- 
cally, the different processes that we have investigated are 
listed below: 

• Extra lines: Here we have added additional lines at 
random locations. The lengths and angles of the added 
lines have been generated by randomly sampling the 
distribution for the existing image-segments. 

• Missing lines: Here we have deleted a known fraction 
of line-segments at random locations. 

• Split lines: Here a predefined fraction of lines have 
been split into two segments. The splitting process is 
effected by deleting an internal fraction of each line- 
segment. The deleted segment is randomly positioned 
along the line. The fraction of the line deleted is uni- 
formly sampled from the range (0, 1). 

• Segment end-point errors: Here we have introduced 
random displacements in the end-point positions for a 
predefined fraction of lines. The distribution of end- 
point errors is Gaussian. The degree of error is con- 
trolled by the variance of the Gaussian distribution. 

• Combined errors: Here we have introduced the four 
different segment errors described above in equal pro- 
portion. 

The performance measure used in our studies is com- 
puted as follows. We query the database with a sample of 
line patterns. For each pattern in turn we determine whether 
or not the correct retrieval occurs in the top-ranked position. 
By computing the fraction of queries that return a correctly 
recognised recall, we determine the average retrieval accu- 
racy. 

We have conducted our experiments with both exact and 
inexact queries. In the former case an exact match to the 
line pattern exists prior to the addition of noise. In the latter 
case, the query pattern is a distorted version of the target in 
the database. We commence by comparing the overall sen- 
sitivity pattern for each of the recognition schemes when 
exact and inexact queries are performed. Figure 2 shows 
the sensitivity plots for the case of exact query, while Fig- 
ure 3 shows the plots for inexact query. Each plot shows the 
recognition accuracy for a particular matching scheme as a 
function of the fi-action of pattern corruption. The different 
curves are for the different noise processes. The main fea- 
ture to note is that in the case of inexact query the shoulder 
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Figure 2. Effect of various kinds of noise for exact queries. 


in the performance curves occurs at a lower noise level for 
each of the recognition strategies. 

Turning our attention first to the sensitivity curves for 
exact query, we make the following observations. Firstly, 
the two histogram based methods (Figure 2(a) and (b)) are 
most susceptible to the addition of extra line-segments. By 
contrast, feature-set (Figure 2(c)) comparison and graph- 
matching (Figure 2(d)) are most susceptible to missing 
lines. 

Since the process of inexact query is more typical of 
large-scale object recognition, we now focus on the asso- 
ciated sensitivity pattern in more detail in Figure 3. 

When retrieval is attempted using graph-matching (Fig- 
ure 3(d)), the following sensitivity pattern emerges. The 
most destructive noise process is the deletion of lines. Here, 
the onset of recognition errors occurs when 10% of lines 
are deleted. Line splitting results in the onset of recognition 
degradation when the error rate is 20%. The technique is 
most robust to the addition of extra lines. Here as many as 
70% of the lines may be added clutter before any degrada- 
tion in recognition performance results. 

In the case of the feature-sets (Figure 3(c)) it is again 
line deletion that proves the most serious obstacle to accu- 
rate recognition. The onset of errors occurs when 40% of 


the lines are deleted. The line-patterns are least sensitive to 
segment end-point errors. In the case of both line-addition 
and line-splitting there is an onset of errors when the frac- 
tion of segment errors is about 20%. However, at larger 
fractions of segmentation errors the overall effect is signif- 
icantly less marked than in the case of line-deletions. The 
technique is considerably less robust to line additions than 
the graph-matching technique. 

In the case of the gated or relational histogram (Fig- 
ure 3(b)), segment end-point error, missing lines and line- 
splitting pose the most serious problems. This is attributable 
to the fact that the line-splitting introduces additional com- 
binatorial background that swamps the query pattern. The 
line-patterns are least sensitive to segment end-point errors. 
In the case of both line-addition and line-splitting there is an 
onset of errors when the fraction of segment errors is about 
20%. However, at larger fractions of segmentation errors 
the overall effect is significantly less marked than in the case 
of line-deletions. Moreover, in each case the recognition 
curves are consistently poorer than either graph-matching 
or feature-sets. 

Finally, the recognition performance of the ungated his- 
togram (Figure 3(a)) is poorest of all for inexact pattern 
matching and is effectively unuseable. Comparing Figures 
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Figure 3. Effect of various kinds of noise for similarity queries. 


2a and 3a it is clear that the transition from exact to inexact 
query has a significant effect on the matching performance. 
To conclude the sensitivity study, we focus more closely 
on the role of segment end-point errors. The reason for this 
is that such errors will effect the accuracy of the relational 
measurements. Figure 4 shows the effect of line end-point 
position errors for inexact queries. The different curves in 
the plots correspond to different values of the standard de- 
viation of the end-point position errors. They show the re- 
trieval accuracy as a function of the fraction of lines affected 
by end-point errors. As the standard deviation of the posi- 
tion error increases, then so the fraction of corrupt lines for 
which perfect recall is possible decreases. The main point 
to note from these plots is that the graph-matching method 
degrades less rapidly under line end-point errors than the 
set-based method and the relational histogram, 

5 Conclusions 

We have compared three different strategies for recall- 
ing structural representations of line-patterns from a large 
database. The main conclusion to be drawn from the study 
is that the best recognition performance is realised when an 
iterative graph-matching scheme is used. This is an inter- 


esting observation, since graph-structure is frequently per- 
ceived as too fragile to be used to noisy information re- 
trieval. There are a number of ways in which the ideas 
presented in this paper can be extended. Firstly, we intend 
to explore more a perceptually meaningful representation 
of the line patterns, using grouping principal derived from 
Gestalt psychology. Secondly, we are exploring the pos- 
sibility of using more complex relational entities as alter- 
natives to edges. Examples include the triangular faces of 
Delaunay graphs [4], Finally, we plan to investigate ways 
on combining the recognition strategies. This would offer 
the advantage that we could take advantage of the comple- 
mentarity of the relational histogram and graph-matching. 
The relational histogram is most robust to missing lines 
while the graph-matching method is most robust to line- 
deletion. There are two ways in which combination might 
be achieved. The first is to use the different similarity mea- 
sures to guide a heuristic search procedure such as the A- 
star algorithm. The second route is to treat each recognition 
strategy as a separate classifier and to combine the decisions 
in an information theoretic framework [11], 
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Figure 4. Effect of introducing segment end-point errors on retrieval performance. 
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